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One way to encourage performance and persistence in STEM fields is to have students write about the 
utility value (UV) or personal relevance of course topics to their life. This intervention has been shown 
to increase engagement and performance in introductory courses. However, questions remain about the 
longevity of the effects and how best to implement the intervention in terms of dosage and timing. We 
tested a UV intervention in the first semester of a 2-semester introductory biology sequence. For each of 
3 units across the semester, students (V = 577) were randomly assigned to receive either a UV writing 
assignment, in which they explained why course material was useful to them personally, or a control 
assignment, in which they summarized course material. This fully crossed design tested the effect of UV 
dosage level (0, 1, 2, or 3 UV assignments) as well as the effect of timing (e.g., UV first, control first). 
We found that students exposed to any dosage of UV earned higher grades in the course, were more likely 
to enroll in the second course of the biology sequence, and were less likely to abandon their STEM major 
than students who did not receive any UV assignments. In terms of timing, students with a history of poor 
performance benefitted from writing a UV essay in the beginning of the semester, whereas higher- 
performing students benefitted from a UV essay at the end of the semester. Recommendations for 
practice are discussed. 


Educational Impact and Implications Statement 

In the current study, we manipulated implementation features of a utility-value (UV) intervention 
(dosage and timing), in which students wrote about the personal relevance and usefulness of course 
topics in an introductory biology sequence for biomedical majors. We found that any exposure to the 
UV intervention increased course performance, continuation to the second course of the biology 
sequence, and persistence within a STEM major. We compared 1, 2, and 3 doses of the intervention 
and found that students who received | or 3 doses benefitted the most from the intervention. 
Furthermore, in terms of timing, students with a history of poor performance earned higher grades 
when they received a single dose of the intervention early in the semester, whereas higher performing 
students were more likely to continue to the second course if they wrote a UV essay at the end of the 
semester. This study addresses key implementation questions for educators who use this intervention 
in their own courses and is the first study to suggest that the positive effects of UV interventions may 
extend beyond a single semester. 


Keywords: academic motivation, interventions, retention, task values, utility value 


High attrition rates in science, technology, engineering, and 
mathematics (STEM) fields have long been a global concern 
(OECD, 2015). A recent report indicated that if the United States 
is to remain competitive in the global economy, it will need 1 


million more STEM professionals over the next decade than it is 
currently projected to produce (President’s Council of Advisors on 
Science and Technology, 2012). Similar shortages in STEM pro- 
fessionals exist in France, Germany, and the United Kingdom 
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(OECD, 2013). To increase the global STEM workforce, it will be 
imperative to retain more college students in STEM majors; how- 
ever, almost half (48%) of students who initially major ina STEM 
field switch into a non-STEM field (Bettinger, 2010; Chen & 
Soldner, 2013; Goulden, Frasch, & Mason, 2009; Higher Educa- 
tion Research Institute, 2010). STEM students typically decide 
whether they will stick with their major within the first two years 
of college and especially after taking introductory STEM courses 
(Seymour & Hewitt, 1997; Strenta, Elliott, Adair, Matier, & Scott, 
1994; Watkins & Mazur, 2013). A number of factors contribute to 
STEM attrition including uninspiring introductory courses and low 
grades in “weed-out” classes, indicating that a positive experience 
in a single introductory course can have an impact on student 
persistence in STEM majors (Gasiewski, Eagan, Garcia, Hurtado, 
& Chang, 2012; Ost, 2010; Seymour & Hewitt, 1997). 

One way to increase performance and retention in introductory 
courses is to highlight the value or relevance of the course mate- 
rial. When students encounter subjects that do not seem important, 
they may become disengaged, lacking the motivation to do well. 
One type of value that has proven to be a powerful predictor of 
persistence and performance is utility value—the perception that a 
task is useful or important for present or future goals (Harackie- 
wicz, Durik, Barron, Linnenbrink-Garcia, & Tauer, 2008; Hulle- 
man, Durik, Schweigert, & Harackiewicz, 2008; Wigfield & Cam- 
bria, 2010). For instance, a student may find a lesson on genetics 
to be important for their personal goal of becoming a doctor or 
because it helps them understand a parent’s disease. By increasing 
perceptions of utility value in introductory STEM courses, educa- 
tors may have the opportunity to influence not only performance in 
the course, but also subsequent course enrollment because of the 
importance of these courses for future goals (e.g., majoring in 
biology, becoming a doctor). If educators can help students find 
the utility value (UV) of course material, they may be able to 
promote achievement and retention in introductory STEM courses 
that act as gateways to successive careers. 


Utility- Value Interventions 


Our approach is based in Eccles’ expectancy-value theory 
(Eccles & Wigfield, 2002), which posits that expectancies for 
success and subjective task values together determine choice, 
persistence, and performance. Eccles identified four types of task 
values (Eccles, 2009; Eccles et al., 1983): intrinsic (the value of 
engaging in the task because it is interesting or enjoyable), utility 
(the perceived importance or usefulness of the task for accom- 
plishing present or future goals), attainment (the perceived impor- 
tance of the task for one’s self concept or identity), and cost (the 
negative aspects of engaging in the task, such as lost time and 
opportunities). A large body of research has examined the role of 
success expectancies or perceived competence (e.g., “I’m really 
good at math and science”) in promoting interest and performance 
in STEM fields (Fredricks & Eccles, 2002; Gaspard, Dicke, 
Flunger, Schreier, et al., 2015; Jacobs, Lanza, Osgood, Eccles, & 
Wigfield, 2002; Watt, 2004). Less focus has been placed on 
subjective task values, which may be more amenable to interven- 
tion, and utility value in particular has proven responsive to ex- 
ternal intervention (Harackiewicz, Tibbetts, Canning, & Hyde, 
2014; Tibbetts, Harackiewicz, Priniski, & Canning, 2016). 


Recent experimental research has focused on promoting percep- 
tions of utility value with the intent of facilitating learning, inter- 
est, and persistence, based on substantial correlational research 
showing that perceptions of utility value predict academic achieve- 
ment, effort, and interest (Hulleman et al., 2008; Wigfield & 
Cambria, 2010). One relatively simple UV intervention instructs 
students to write about the relevance or utility value of course 
topics to their own life (Hulleman & Harackiewicz, 2009). The UV 
intervention as implemented in previous research (as well as the 
current study) targets both relevance and utility value. Although 
these constructs are theoretically distinguishable—relevance is 
defined as a personally meaningful connection to the individual, 
whereas utility value is a type of relevance focused on personal 
usefulness (see Priniski, Hecht, & Harackiewicz, in press)—in 
practice, similar interventions grounded in the same theoretical 
model have been called relevance interventions or utility-value 
interventions (e.g., Hulleman, Godes, Hendricks, & Harackiewicz, 
2010; Hulleman & Harackiewicz, 2009; Gaspard, Dicke, Flunger, 
Brisson, et al., 2015). Students may write about relevance or 
personal usefulness (or both) and research to date does not distin- 
guish between these processes. Regardless, a goal of these inter- 
ventions is to help students perceive and articulate the utility value 
of the material they are learning. In line with this goal and with the 
theoretical underpinnings of the intervention, we call the interven- 
tion a utility-value intervention. 

The utility-value writing intervention has been shown to work 
best for high school students who doubt their competence and for 
college students with a history of poor performance (Canning & 
Harackiewicz, 2015; Gaspard, Dicke, Flunger, Brisson, et al., 
2015; Harackiewicz, Canning, Tibbetts, Priniski, & Hyde, 2016; 
Hulleman et al., 2008; Hulleman et al., 2010; Hulleman & Har- 
ackiewicz, 2009; Hulleman, Kosovich, Barron, & Daniel, 2017). 
For example, Hulleman and Harackiewicz (2009) implemented a 
UV intervention for high school students by having students write 
about the personal relevance of their science schoolwork. They 
found that students with less confidence in their science class 
reported higher interest and improved their grades in the UV 
condition, relative to a control condition, whereas confident stu- 
dents made no significant gains with the intervention. Hulleman et 
al. (2010) implemented the same intervention in a college intro- 
ductory psychology class and found that students who had per- 
formed poorly on an early exam and who wrote about UV reported 
more interest in the course at the end of the semester when 
compared to those in the control group. More recently, a UV 
writing intervention promoted performance for underrepresented 
students in a college introductory biology course (Harackiewicz, 
Canning, et al., 2016). The UV intervention was successful in 
reducing the achievement gap for underrepresented minority stu- 
dents by 40% and for underrepresented minority students who 
were also first-generation college students by 61% (Harackiewicz, 
Canning, et al., 2016). Harackiewicz, Canning, et al. (2016) also 
found that the UV intervention was most effective for students 
who had the lowest prior GPAs, indicating that the intervention 
helped students with a history of poor performance, in addition to 
helping underrepresented minority students. 

Across studies the UV intervention has been found to be most 
effective for students who are in most need of intervention. In 
laboratory studies and studies with younger students, an important 
moderator of intervention effects is success expectancies (confi- 
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dence about performance), such that the intervention is most 
effective for students with low confidence; whereas in studies with 
college students, prior performance is a more critical moderator, 
such that the intervention is most effective for students with a 
history of poor performance. Indeed, Harackiewicz, Canning, et al. 
(2016) tested both performance (prior GPA) and success expec- 
tancies as moderators of a UV intervention implemented in a 
college biology course and found that the intervention effects were 
moderated by performance and not by success expectancies. 


Retention Outcomes 


UV writing interventions have been shown to be effective at 
increasing task engagement and academic performance within a sin- 
gle course or specific context; however, it is unknown whether inter- 
vention effects can extend over time. Little research has investigated 
the impact of UV interventions on retention, course-taking, and major 
persistence. Expectancy-value theory suggests that when students 
perceive utility value in their course, they are by definition finding 
value for a future situation or personal goal. Perceived utility value for 
one task or topic (e.g., a single introductory course) should facilitate 
persistence in achieving future goals. Therefore, UV writing interven- 
tions have the potential to impact retention outcomes, such as remain- 
ing in a major or persisting in a difficult sequence of introductory 
courses (Harackiewicz, Smith, & Priniski, 2016). However, existing 
studies of UV writing interventions have not investigated the possi- 
bility of long-term impacts beyond the course in which the interven- 
tion was implemented. To address this gap in the research, we inves- 
tigate whether a UV writing intervention can influence outcomes such 
as course taking and major persistence. 


Intervention Implementation: Dosage and Timing 


UV writing interventions have already proven successful at in- 
creasing a number of positive outcomes; however, important ques- 
tions remain about how best to implement the intervention. First, what 
is the optimal dosage for these writing assignments? The number of 
writing assignments has varied across studies. In high school science 
courses, the dosage ranged from 1—8 assignments over the course of 
a semester (Hulleman & Harackiewicz, 2009). In high school math 
courses, the primary intervention was followed by two short rein- 
forcement assignments, | and 2 weeks after the intervention (Gaspard, 
Dicke, Flunger, Brisson, et al., 2015). In college classes, Hulleman et 
al. (2010) implemented a UV writing intervention twice during the 
second half of the semester (weeks 10 and 12 of a 15-week semester). 
Hulleman et al. (2017) also administered two doses of the writing 
intervention, but did so in the first half of the semester (weeks 4 and 
8 of a 15-week semester). Additionally, UV writing assignments have 
been implemented three times during a semester, once for every major 
unit in the course (Harackiewicz, Canning, et al., 2016). It is unclear 
how many writing assignments are necessary for the intervention to 
be most effective. 

Based on expectancy-value theory, we might expect that the more 
times a student is given the opportunity to discover the utility value of 
course material, the more likely they are to perceive value and thus 
exhibit greater motivation. In one study, Hulleman et al. (2017) found 
that the self-reported frequency with which students made connec- 
tions between the course material and their lives was positively related 
to perceptions of utility value and continued interest in the course. In 


other words, students may benefit from multiple opportunities to 
make UV connections with the material, and frequent UV writing 
assignments could be a powerful tool to create those opportunities. 
Therefore, it is possible that the optimal dosage involves administer- 
ing UV writing assignments repeatedly, for each unit or topic of the 
course. Considering that the UV intervention is a curricular assign- 
ment based in course content, students may need to be prompted to 
think about the utility for each unit or topic of the course. However, 
this has not been tested systematically, and it is possible that a smaller 
number of UV writing assignments could be just as effective. For 
example, a single UV assignment might be powerful because of its 
novelty in a science class. An alternative hypothesis is that too many 
UV writing assignment might be unnecessary (e.g., if a single assign- 
ment is sufficient to promote value) or even become overly burden- 
some to students and faculty, reaching a point of diminishing returns. 

Practically speaking, it is important to investigate whether multiple 
UV writing assignments is the best practice, considering the time that 
instructors invest in grading these assignments and providing written 
feedback for students. In past implementations, the UV essay assign- 
ments were graded for accuracy of scientific content and students 
received feedback in the form of individualized written commentary 
(Harackiewicz, Canning, et al., 2016). If this intervention is to be 
scaled up and implemented broadly, it needs to be efficient and 
cost-effective. The UV exercise is costly in terms of time commit- 
ment, both for students (who write the essays) and for instructors 
(who grade them); however, this cost is justified if students benefit 
from the intervention. It is important to know the minimum number of 
UV exercises required to produce maximum effectiveness. 

Another implementation question relates to the timing of the writ- 
ing assignments: Should educators incorporate UV assignments at the 
beginning or end of semester? It is not well understood when UV 
information is most effective. Research from the field of intervention 
science suggests that interventions should be administered at critical 
times in a student’s development to elicit recursive processes, such as 
the beginning of the school year or the beginning of a difficult course 
(Cook, Purdie-Vaughns, Garcia, & Cohen, 2012; Harackiewicz & 
Priniski, 2018; Yeager & Walton, 2011). Through recursive pro- 
cesses, a well-timed intervention initiates small psychological effects, 
such as perceptions of utility value, that then gain momentum over 
time, eventually leading to adaptive academic behaviors. In that 
regard, the UV intervention promotes deeper engagement with course 
material, and setting this process into motion early could change the 
way students approach subsequent material. 

The timing of the UV intervention may be even more critical for 
students who doubt their competence or for those with a history of 
poor performance. Early implementation of the UV intervention 
may be critical for low-performing students to increase engage- 
ment and performance in the course, whereas implementing the 
UV intervention later in the semester might be critical for course- 
taking and major decisions. Implementing the intervention early in 
the semester may facilitate learning and provide a reason for 
students to engage with the course material. For instance, a single 
UV assignment, administered early in the semester, may establish 
a “habit of mind” so that students continue to make UV connec- 
tions with the course material without needing to write more UV 
essays (Hulleman et al., 2017). On the other hand, students may 
need more time to absorb the course material before UV informa- 
tion can influence course-taking and major decisions, and there- 
fore, assigning UV writing assignments later in the semester may 
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be more effective for these outcomes. Thus, timing effects may 
depend on two things: students’ prior achievement, with early 
intervention hypothesized to be more effective for students who 
struggle, and the outcome measure, with early intervention hypoth- 
esized to be more effective for performance in the course, and later 
intervention more important for decisions about the future. The 
existing literature contains examples of UV interventions imple- 
mented early in the semester (Hulleman et al., 2017), later in the 
semester (Hulleman et al., 2010), and throughout the semester 
(Harackiewicz, Canning, et al., 2016; Hulleman & Harackiewicz, 
2009), but no study has directly compared these implementation 
strategies to test whether the timing of the intervention moderates 
its effectiveness. The design and large sample of the current study 
allowed us to test timing and dosage effects for UV interventions. 


The Present Study 


We tested a UV writing intervention in the first semester of a 
two-semester introductory biology course. We address two primary 
questions about the UV intervention: (1) Can the UV intervention 
increase performance as well as retention outcomes, such as course- 
taking and deciding to remain in a STEM field? and (2) What is the 
optimal timing and dosage for the UV writing intervention? Research 
to date is not conclusive regarding the number of UV essays to assign 
in a course, or whether this intervention is more effective earlier or 
later in a course. We varied the number (dosage level) and sequence 
(timing) of UV writing assignments, by varying whether the first, 
second, and third essay assignments were UV or control. This design 
was fully crossed, resulting in a 2 (First Essay: UV vs. Control) by 2 
(Second Essay: UV vs. Control) by 2 (Third Essay: UV vs. Control) 
design. These eight experimental conditions allowed us to test the 
effect of UV dosage level (0, 1, 2, or 3 UV essays) as well as the effect 
of timing (by comparing groups who write UV assignments early in 
the semester with those who write them later). 

To address our first research question, we tested the efficacy of 
the UV intervention averaged across dosage and timing. We hy- 
pothesized that the UV intervention would be effective at increas- 
ing performance as well as retention in the biology course se- 
quence and persistence within a STEM major. We also 
hypothesized that increased grades in the course would mediate the 
effect of the intervention on retention and STEM major persis- 
tence. In other words, we expected that the UV intervention would 
lead students to continue in the biology course sequence and 
remain in a STEM major by first increasing their grades in the 
course. To address our second research question, we conducted 
three separate analyses to determine the optimal dosage and timing 
of the UV intervention. First, we tested the fully crossed 2 (First 
Essay: UV vs. Control) by 2 (Second Essay: UV vs. Control) by 2 
(Third Essay: UV vs. Control) model. By testing the fully crossed 
model, we were able to examine effects of timing and dosage 
simultaneously. In terms of optimal dosage, we hypothesized that 
the UV intervention would be most effective at full dosage (three 
times throughout the semester or one UV assignment for each unit 
of the course). In terms of optimal timing of the intervention, 
however, our hypotheses were more exploratory, and we also 
examined the effects of timing in more fine-grained follow-up 
analyses. Specifically, we focused on a subset of the design where 
we could examine the effect of a single dose of UV writing, 
administered at different time points, and hypothesized that a 


single UV writing assignment would be more effective at the 
beginning of the semester compared with the end of the semester. 
In all analyses we included one moderator (prior GPA), and two 
covariates: gender and initial interest in biology. Prior GPA was 
chosen as a moderator because prior research has shown that the UV 
intervention was more effective for students with a history of poor 
performance (Hulleman & Harackiewicz, 2009; Hulleman et al., 
2010, 2017). We hypothesized that the UV intervention would be 
more effective at increasing grades for students with lower prior 
GPAs as in previous research. In terms of timing, we hypothesized 
that early implementation of the UV intervention and multiple dosage 
of the interaction would be most beneficial for low-performing stu- 
dents. In this biology course men typically outperform women, so it 
was important to control for this effect in the analyses. Additionally, 
it is important to control for baseline performance when examining 
treatment effects on course performance and baseline interest when 
examining treatment effects on course-taking and major decisions. 


Method 


We implemented the UV intervention in a double-blind, random- 
ized experiment in three sections of an introductory biology course at 
a large Midwestern university. The course is the first of a two- 
semester sequence required for 34 undergraduate majors in the bio- 
logical and medical sciences (e.g., zoology, biochemistry, neurosci- 
ence, nursing). This important gateway course covers three units— 
cellular biology, genetics, and evolutionary biology—and consists of 
lectures, laboratory sections, and discussion sections, for a total of 5 
credits. The course is team-taught, with a different professor for each 
topic and teaching assistants leading the laboratory and discussion 
sections. Of the 598 students eligible to participate in the study, 577 
students (247 male, 330 female) completed the course and gave 
consent for access to their academic records (3 students did not 
consent; 18 dropped the course).' Participants were 86% White and 
14% Asian/Asian American. Preliminary analyses revealed that there 
were no significant differences between White and Asian/Asian 


'There were 241 other students enrolled in this course who were not 
eligible to participate in this study: 161 first-generation (FG) majority college 
students, 50 underrepresented minority (URM) students, and 30 students who 
were both FG and URM. These students participated in a different study, with 
a different experimental design, intended to test the efficacy of the UV 
intervention at full dosage to close achievement gaps (Harackiewicz, Canning, 
et al., 2016). It is important to note that all students in this class were treated 
identically (same assignments, grading structure, and course content); the only 
difference between the sample presented here and the one in Harackiewicz, 
Canning, et al. (2016) is which analytic design the students were assigned to. 
Harackiewicz, Canning, et al. (2016) employed a 2-cell UV intervention 
design across 4 semesters and the current study employed an 8-cell UV design 
in a single semester. Harackiewicz, Canning, et al. (2016) found that the UV 
intervention improved grades for FG-URM students. In that study, the UV 
intervention was crossed with a values-affirmation intervention, but there were 
no significant effects of the VA intervention. In the current study, an 8-cell UV 
intervention design was crossed with the same VA intervention (treatment vs. 
control), resulting in a 16-cell design. Full methodological details regarding 
implementation and testing of the VA intervention in this class are reported by 
Harackiewicz, Canning, et al. (2014) and Harackiewicz, Canning, et al. (2016). 
However, there were no significant effects of the VA intervention or any 
interactions with the UV intervention in the present study, and we therefore 
collapsed across VA condition for all analyses reported here. In addition, we 
tested the efficacy of the UV intervention in the control condition of the VA 
intervention, and found comparable effects of the UV intervention on grades, 
continuation, and STEM major persistence. 
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American students or any interactions with the UV intervention on 
any outcome measure. Therefore, White and Asian/Asian American 
students were combined for analyses. The researchers’ Institutional 
Review Board approved this study. 


Utility-Value Intervention 


Students were blocked on gender and randomly assigned to one 
of 8 conditions testing dosage and timing of the UV intervention. 
For each of the 3 5-week units of the course, students were 
assigned a UV or control writing assignment, in a fully crossed 2 
(First Essay: UV vs. Control) by 2 (Second Essay: UV vs. Control) 
by 2 (Third Essay: UV vs. Control) design. 

Both the UV and control assignments instructed students to: 


Select a concept or issue that was covered in lecture and formulate a 
question. Select the relevant information from class notes and the 
textbook, and write a 1-2 page essay. 


The UV assignment then instructed students to answer their 
question using course material and were given the choice to 
discuss the relevance of the concept or issue to either their own or 
others’ lives: 


Write a 1-2 page essay addressing this question and discuss the 
relevance of the concept or issue to your own life. Be sure to include 
some concrete information that was covered in this unit, explaining 
why this specific information is relevant to your life or useful for 
you. Be sure to explain how the information applies to you personally 
and give examples. 


OR 


Write a 1-2 page letter to a family member or close friend, addressing 
this question and discuss the relevance of this specific concept or issue 
to this other person. Be sure to include some concrete information that 
was covered in this unit, explaining why the information is relevant 
to this person’s life, or useful for this person. Be sure to explain 
how the information applies to this person and give examples. 


Consistent with previous implementations of the UV interven- 
tion in college classes (Hulleman et al., 2010; Harackiewicz, 
Canning, et al., 2016), we gave students the choice in UV condi- 
tions to discuss how course content is relevant to either their own 
or others’ lives, hypothesizing that students who have a difficult 
time connecting the material to their own life may be able to 
generate utility value for someone else’s life more easily than their 
own. The control assignment instructed students to answer their 
question using course material: 


Select the relevant information from class notes and the textbook, and 
write a 1-2 page response to your question. You should attempt to 
organize the material in a meaningful way, rather than simply listing 
the main facts or research findings. Remember to summarize the 
material in your own words. 


The UV or control assignments were assigned three weeks 
before each unit exam. Students were given five days to complete 
the homework assignment and turn it in to a dropbox on the course 
management website. Biology graduate students graded the assign- 
ments for scientific content, writing quality, and whether students 
followed directions. To ensure that instructors and graduate teach- 
ing assistants remained blind to experimental condition, the as- 


signments were emailed to students by course coordinators, and 
any graduate students who served as both teaching assistants and 
graders in the course were assigned to grade essays for a different 
section, so that no teaching assistant ever graded their own stu- 
dents’ essays. 

Grades and grader feedback were provided to the students a few 
days before the unit exam. Each essay was worth 0.6% of the 
students’ final grade; 574 students completed the first assignment, 
567 students completed the second assignment, and 555 students 
complete the third assignment, with 550 students (95.3%) com- 
pleting all three assignments. 


Baseline Measures 


Questionnaire measures of attitudes about biology were col- 
lected in laboratory sections during the second week of the course. 
Questionnaire items were answered on a 7-point Likert-type scale 
(1 = not at all true, 7 = very true or | = not at all, 7 = a lot), 
unless otherwise noted. Scale scores represent the mean of con- 
stituent items. Missing data (less than 1% on each measure) were 
imputed using SPSS Missing Values software (version 24) with 10 
imputed data sets (Rubin, 1987). 

Initial interest in biology. Interest in biology was measured 
with five items (“I’m really looking forward to learning more 
about biology,” “Biology fascinates me,” “I think the field of 
biology is very interesting,” “I’m excited about biology,” “To be 
honest, I just don’t find biology interesting,” reversed, a = .93). 
Questionnaire measures of interest typically include items that 
measure value as well as positive affect (Harackiewicz et al., 2008; 
Renninger & Hidi, 2011; Schiefele, 1991); however, given that the 
UV intervention targets values explicitly, we focus on affective 
processes in order to separate interest from value in our measure- 
ment. For example, we did not include two items from an earlier 
measure of initial interest (“I think what we will study in this 
course will be important for me to know,” and “I think what we 
will study in this course will be worthwhile to know”; Harackie- 
wicz et al., 2008). 

Prior performance. We obtained students’ prior college GPA 
from university records (n = 533). For students with missing 
records (e.g., transfer students, first-year students), we used stu- 
dents’ self-reported college GPA from the baseline questionnaire 
(n = 29). Cases with missing data on all prior performance 
measures (n = 15) were imputed using SPSS Missing Values 
software (version 24) with 10 imputed data sets (Rubin, 1987). 


Outcome Measures 


Biology course grade. Course instructors provided final 
course grades at the end of the semester, on a 4.0 scale (A = 4.0, 
AB = 3.5, B = 3.0, BC = 2.5, C = 2.0, D = 1.0, F = 0). Grading 
standards and procedures were criterion based and consistent 
across sections (M = 2.94, SD = .72). 

Continuation to the second course of the introductory biol- 
ogy sequence. We assessed students’ course taking in the fol- 
lowing semester, using course records, to determine whether they 
continued on to the second course of the introductory biology 
sequence. Though it is possible (but quite rare) for students to take 
the second course nonconsecutively, both semesters are required 
for biological and medical science majors. Failure to enroll in the 
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second course would exclude students from 34 biological and 
medical majors and is therefore a preliminary indicator that a 
student is moving away from the biomedical sciences. 

STEM major persistence. Self-reported college major was 
measured at the beginning and end of the semester. If students had 
not formally declared a major (n = 270; 46.8% of the sample), 
they were asked to list the major(s) they were considering. Majors 
were coded as STEM or non-STEM by two coders with an inter- 
rater reliability of 95%. For the purposes of this study, majors in 
the physical, biological and natural sciences, engineering, com- 
puter science, and mathematics were coded as STEM majors. 
Majors in the social sciences and humanities were coded as non- 
STEM. We compared students’ reported major at the beginning of 
the semester with their major at the end of the semester and 
classified students into one of four categories: remained in non- 
STEM major, changed from non-STEM to STEM major, remained 
in STEM major, changed from STEM to non-STEM major. Be- 
cause the majority of students entered the course with a STEM 
major (92.5%), we focus primarily on whether students remained 
in STEM or changed their intended major to a non-STEM field at 
the end of the semester, as a measure of STEM major persistence. 
Eleven students did not report their major at either the beginning 
and/or the end of the semester and were therefore excluded from 
analyses on this variable. 

Articulated utility value. The UV and control writing assign- 
ments were coded for the level of utility value articulated in the 
essay. Research assistants coded the assignments on a 0—4 scale 
based on how specific and personal the UV connection was to the 
individual (i.e., the student or a close friend or family member for 
students who chose to write a letter in UV conditions). A “0” on 
this scale indicates no utility or relevance; a “1” indicates general 
utility or relevance applied to humans generically; a “2” indicates 
utility or relevance that is general enough to apply to anyone, but 
is applied to the individual; a “3” indicates utility or relevance that 
is specific to the individual; and a “4” indicates a strong, specific 
connection to the individual that includes a deeper appreciation or 
future application of the material. UV scores from the three essays 
were averaged to create an overall measure of articulated utility 
value, with higher scores indicating higher-quality UV connec- 
tions. Interrater reliability with this coding rubric was high, with 
two independent coders providing the same score on 91% of 
essays. Disagreements were resolved by discussion. 
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Results 


Analytic Strategy 


Table 1 presents descriptive statistics and intercorrelations for 
all measures. Although students were nested within three lecture 
sections, randomization to condition occurred at the student level. 
The intraclass correlation coefficient was small; lecture sections 
accounted for only 2.4% of the variance in biology course grade. 
Indeed, hierarchical linear model (HLM) and regression yielded 
consistent results. Therefore, all analyses reported here were con- 
ducted with regression so that we can report effect sizes (betas). 

The primary outcome measures were the final grade in the 
biology class, continuation in the introductory biology sequence, 
and STEM major persistence across the semester. Articulated 
utility value was examined as a check of the UV manipulation. We 
used OLS regression for the continuous outcome measure (biology 
course grade and articulated utility value) and logistic regression 
for the dichotomous outcome measures (continuation into the 
second course and STEM major persistence). Significant effects 
are described below, and full regression results are displayed in 
Table 2. 


Test of Intervention Efficacy 


First, we tested the efficacy of the UV intervention averaged 
across dosage and timing. We tested the main effect of the UV 
intervention (control = —1, UV = 1), collapsing across 7 UV 
conditions and controlling for gender (female = —1, male = 1), 
prior performance, and initial interest in biology. We also tested 
the interaction of the UV intervention with prior performance, 
allowing us to test for replication of prior UV research that found 
that the UV intervention was particularly effective for students 
with low grades (Harackiewicz, Canning, et al., 2016; Hulleman et 
al., 2010). In preliminary analyses, we tested initial interest in 
biology and gender as moderators of the UV intervention, however 
there were no interactions with the UV intervention on any depen- 
dent variable for either moderator, therefore we trimmed these 
interactions from the final model. 

Manipulation check: Articulated utility value. To test 
whether the UV intervention led participants to articulate more 
utility-value connections in their essays than those in the control 


Zero-Order Correlations and Descriptive Statistics for Major Variables 


Variable 1 2 3 4 5 6 
1. Prior GPA _ 
2. Articulated utility value 10° — 
3. Biology course grade .60°* .10* —_— 
4. Continuation to second course A5”* 04 28" — 
5. STEM major persistence .09 04 14" 22" — 
6. Initial interest in biology —.01 —.04 Av 2 .03 — 
Range 86-4 0-4 0-4 — —_ 1-7 
M or % 3.27 1.74 2.94 78.3% 92.5% 5.82 
SD 54 1.00 12 —_ — 1.00 
N 577 S77 577 577 534 S77 
Note. GPA = grade-point average; STEM = science, technology, engineering, and math. 


“yp <.05. “p<.0l. 
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Table 2 


CANNING ET AL. 


Effects of the UV Intervention on Course Grade, Continuation, and STEM Major Persistence 


Continuation to second STEM major 
Biology course grade course persistence 
Predictor p (571) Dp B Wald Pp B Wald Dp 
UV vs. Control 09 2.63 .009 30 4.55 .033 OL 4.44 035 
Prior GPA 62 13:27 .000 20 3.46 .063 44 4.39 .036 
UV X Prior GPA —.02 —.34 734 AT 1.50 221 —.08 AS 696 
Gender .10 3.03 .003 =13 1.45 .229 16 58 445 
Initial interest in biology A 3339 .001 2 25.40 .000 18 75 387 
Note. UV = utility-value; UV vs. Control (control = —1, UV conditions = 1); Gender (female = —1, male = 


1). GPA = grade-point average. 


condition, we examined articulated utility value, which was mea- 
sured on a 4-point scale assessing the quality of the utility value 
articulated in an essay. As expected, students in the UV conditions 
made higher-quality UV connections (i.e., made more personal and 
more specific connections to curricular content) in their essays 
(M = 1.96, SD = .91) than those in the control condition (M = .45, 
SD = .40), (571) = 14.78, B = 0.52, p < .001.” Indeed, the 
majority of students in the control condition (88%) made no 
personal UV connections in any of their essays. This important 
manipulation check indicates that the UV intervention was suc- 
cessful in encouraging students to make personal connections with 
the course material in their writing assignments. Gender, initial 
interest in biology, Prior GPA, and the interaction between the UV 
intervention and prior GPA, were not significant. 

Biology course grade. We found that the UV intervention 
increased course grades (M = 2.96, SD = .71), compared with the 
control condition (M = 2.77, SD = .75), B = .09, p = .009, 
indicating that students who wrote a UV essay at any point in the 
semester received on average a higher grade in the course than 
students in the control group. There was a positive effect of 
gender, B = .10, p = .003, indicating that men performed better 
than women in this course. Prior GPA, B = .62, p < .001, and 
initial interest in biology, B = .11, p = .001, were significant 
predictors of biology course grade, indicating that students with 
higher prior GPAs and higher interest at the beginning of the 
course received higher grades in the course. The interaction be- 
tween the UV intervention and prior GPA was not significant. 

Continuation to the second course. We tested the same 
model used for biology grade, but used binary logistic regression 
to examine whether students enrolled in the second course of the 
biology sequence. Students in the UV condition were more likely 
to enroll in the second course of the sequence, Wald = 4.55, B = 
30, p = .033. We found that 79.80% of students who wrote about 
UV at least once continued to the next semester, compared to only 
69.50% in the control condition. Initial interest in biology signif- 
icantly predicted whether students enrolled in the second course of 
the sequence, Wald = 25.40, B = .52, p < .001. Prior GPA was 
a marginal predictor of enrollment in the second course of the 
sequence, Wald = 3.46, B = .25, p = .063; however, this effect 
did not reach significance. The interaction between the UV inter- 
vention and prior GPA was not significant and there was no effect 
of gender. 

STEM major persistence. We used binary logistic regression 
to test whether the UV intervention influenced students’ STEM 


major decisions at the end of the semester using the same model as 
above. Our primary interest is whether students persisted in a 
STEM major across the semester, and so we excluded students 
who were not STEM majors when they started the class. There- 
fore, this analysis is conducted with the 534 students (93%) who 
initially reported a STEM major at the beginning of the semester. 
Among these students, we found that students in the UV conditions 
were more likely to persist in their STEM major, Wald = 4.44, 
B = 51, p = .035 (see Table 3). That is, 96.1% of students in the 
UV conditions remained in a STEM major from the beginning to 
the end of the semester, compared with 89.2% of students in the 
control condition. This suggests that the UV intervention was 
successful at retaining students in STEM disciplines across the 
semester, relative to the control condition. Prior GPA was a 
significant predictor of STEM major persistence, Wald = 4.39, 
B = A4, p = .036. The interaction between the UV intervention 
and prior GPA was not significant and there was no effect of 
gender or initial interest in biology. 

Mediation analyses. We used Hayes’ (2013) bootstrapping 
procedure with PROCESS software, which allowed us to test 
biology course grade as a mediator of the UV intervention effect 
on enrollment in the second course of the sequence and STEM 
major persistence (see Figure 1). We hypothesized that the UV 
intervention would increase grades in the course, leading students 
to be more likely to enroll in the second course of the sequence and 
persist in their STEM major. Results based on 5,000 bootstrapped 
samples indicate that the indirect effect via biology course grade 
equaled .08, 95% CI [.0172, .1781] for enrollment in the second 
course and .06, 95% CI [.0007, .1685] for STEM major persis- 
tence. The confidence intervals of the indirect effects do not 
include zero; thus we can conclude that biology course grade 
partially mediated the positive effect of the UV intervention on 
continuation to the second course of the sequence and STEM 
major persistence. 


? We also tested whether articulated value differed between the two 
types of UV assignment (i.e., when students chose to write about utility for 
themselves vs. for others) on the first essay. We found that students’ 
assignments contained somewhat higher-quality utility-value connections 
when they wrote about utility value for others than when they wrote about 
utility value for themselves, p = .001. However, the difference between the 
two UV assignments (.43) was smaller than the difference between UV 
assignments and control assignments (2.57). 
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Table 3 
Reported Major From Beginning to End of Semester 
UV 
Major Control conditions Total 
Started in non-STEM major 
Remained in non-STEM 2 (40.0%) 19(70.4%) 21 
Changed from non-STEM to STEM = 3 (60.0%) 8 (29.6%) ila 


Total 5 27 32 
Started in STEM major 


Remained in STEM 66 (89.2%) 442 (96.1%) 508 


Changed from STEM to non-STEM 8 (10.8%) = 18 (3.9%) 26 

Total 74 460 534 
Note. UV = utility value; STEM = science, technology, engineering, 
and math. 


We also tested whether articulated utility value (measured be- 
fore the end of the course) explained the UV intervention effect on 
course grades, continuation, and STEM major persistence; how- 
ever, these indirect effects were not significant. The indirect effect 
via articulated utility value equaled .02, 95% CI [-.0252, .0604] for 
biology course grade, —.06, 95% CI [-.2465, .1260] for continu- 
ation to the second semester, and —.16, 95% CI [-.6122, .2710] for 
STEM major persistence. 


Test of Intervention Timing and Dosage: 
Crossed Design 


We found that the UV intervention increased grades, promoted 
continuation in the biology course sequence, and promoted reten- 
tion for students in STEM majors across the semester; however, an 
important question remains about how best to implement UV 
interventions: What is the optimal timing and dosage of the UV 
assignments? To address this research question, we tested whether 
the UV assignments were more effective at different points in the 
semester and whether there was value added by assigning more 
UV essays in the course. Utilizing the 2 (First Essay: UV vs. 
Control) by 2 (Second Essay: UV vs. Control) by 2 (Third Essay: 
UV vs. Control) fully crossed design, we tested all interactions 
between the three treatment main effects (control = —1, UV = 1; 
for each of the three essays). We also tested all interactions with 
prior GPA, allowing us to examine whether replication of prior UV 
research (Harackiewicz, Canning, et al., 2016; Hulleman et al., 
2010) depends on the timing and dosage of the intervention. 
Gender (female = —1, male = 1) and initial interest in biology 
were included as covariates. The final model included 18 terms: 
the main effects of the first, second, and third essay, prior GPA, six 
two-way interactions, four three-way interactions, one four-way 
interaction, and two covariates (see Table 4 for descriptive statis- 
tics by experimental condition and Table 5 for full regression 
results). The primary outcome measures for the test of timing and 
dosage were articulated utility value, the final grade in the biology 
class and continuation in the introductory biology sequence (given 
that 92.5% of students who reported majoring in STEM at the 
beginning of the semester persisted with a STEM major by the end 
of the semester, we did not have enough power to test the fully 
crossed model on STEM persistence). To interpret significant 
interactions with prior GPA, predicted values were generated for 
individuals one standard deviation below and above the mean. 


Manipulation check: Articulated utility value. As expected, 
we found a treatment effect for each time point: first essay, B = 
44, p < .001, second essay, B = .44, p < .001, third essay, B = 
45, p < .001, indicating that students writing a UV essay at any 
time point, made higher-quality UV connections in their essays, on 
average, compared to writing a control essay at each time point. 
This pattern of findings suggests an additive effect of UV dosage, 
such that three doses might prove optimal. Indeed, post hoc anal- 
yses using LSD indicate that students made higher-quality UV 
connections, on average, when they received three doses of UV, 
compared to one or two doses of UV (see Table 4). To examine 
dosage effects on articulated utility value, it is important to con- 
sider that some students wrote more UV essays than others, given 
our experimental design, therefore, we report supplementary anal- 
yses below to examine these effects in greater detail. 

There was a significant two-way interaction with the first and 
second essay, B = .05, p = .039, indicating that there was a 
synergistic effect of writing a UV essay for the first two units of 
the course. In other words, students made higher-quality UV 
connections when they wrote a UV essay for the first two units 
(Y = 2.70), compared to writing a UV essay for either the first 
(Y = 1.70) or second unit (Y = 1.70) only, or a control essay for 
both units (Y = 0.92). There was also a marginal interaction 
between the third essay and prior GPA, B = .05, p = .056, 
indicating that students with high prior GPAs made somewhat 
higher-quality UV connections in UV essays for the third unit of 
the course (Y = 2.29), compared with students with low prior 
GPAs (Y = 2.11), though this difference was relatively small. 
Gender, initial interest in biology, and Prior GPA were not signif- 
icant. 

Biology course grade. We tested the fully crossed model to 
examine whether the UV assignments were more effective at 
different points in the semester. First, we examined the UV con- 


Biology Course 


Grade 
.09"* 93" 
(.03 (.19) 
Utility Value vs. .30* (.14) Continuation to 
Control .22 (.14) Second Course 
Biology Course 
Grade 
.09** 52 
(.03 (.34) 
Utility Value vs. 51° (.24) STEM Major 
Control AS (.25) Persistence 
Figure I. Mediation model showing the effect of the utility-value inter- 


vention on continuation to the second course in the sequence and STEM 
major persistence, as mediated by biology course grade. Values represent 
unstandardized coefficients, and values inside parentheses represent stan- 
dard errors. Regression analyses include gender, initial interest in biology, 
and prior GPA as covariates. Along the lower path, the values above the 
arrow show the total effect of the utility-value intervention, and the values 
below the arrow show the direct effect. * p < .05, ““ p < .01, “““ p < .001. 


° 
Q 
a 
a 
< 
(3) 
bk 
io} 
° 
aq 
2. 
> 
an 
Ay 
(3 
3) 
< 
o 
cg 
mal 
Oo 
no) 
2 
cas 
ob 
z 
° 
is) 
a 
3) 
is) 
& 
=) 
a 
— 


e of its allied publishers. 


4) 


adly. 


and is not to be disseminated bro 


This article is intended solely for the personal use of the individual user 


842 


Table 4 


CANNING ET AL. 


Descriptive Statistics for Course Grade, Continuation, and Articulated Utility Value by Experimental Condition 


Utility value Unit for utility value 


Biology course Continuation to Articulated utility 


dosage assignment Label N grade M (SE) second course % value M (SE) 

0 None CCC 82 2.78 (.06)* 69.98)! 42 (.07)” 

1 Ist VCC 71 3.00 (.07)°° 85.375" 1.26 (.07)* 
2nd CVC 70 2.97 (.07)°* 81.84™ 1.26 (.07)* 
3rd CCV 71 2.98 (.07)°° 84.408" 1.39 (.07)* 

2 Ist & 2nd VVC 71 2.91 (.07)#° 64.32! 2.27 (.07)” 

Ist & 3rd VCV 67 2.83 (.07)” 75.67" 2.16 (.08)” 
2nd & 3rd CVV 69 2.97 (.07)°* 75.61)! 2.15 (.07)* 

3 All VVV 76 3.05 (.06)° 89.80™ 3.14 (.07)” 

Note. Predicted values from the regression equation, at the mean of each covariate (prior GPA, interest, gender). Post hoc comparisons using LSD are 


shown using superscripts; means/percentages with different superscripts are different at the p < .05 level. C = Control; V = Utility Value. 


dition main effects for essay 1, essay 2, and essay 3, to determine 
whether there were unique benefits to receiving a UV assignment 
at any of the three time points. None of the treatment main effects 
were significant, p > .10. In other words, the UV assignments 
were not differentially effective at the three time points. However, 
there was a significant three-way interaction between the three UV 
condition terms, B = .09, p = .006. To probe the meaning of the 
three-way interaction, we conducted post hoc comparisons using 
LSD (see Table 4) and discuss overall patterns below. There was 
a positive effect of gender, 8 = .10, p = .004, indicating that men 
performed better than women in this class. Prior GPA, B = .61, 
p < .001, and initial interest in biology, B = .12, p = .001, were 
also significant predictors of biology course grade. There were no 
significant interactions with prior GPA. 

Continuation to the second course. In contrast to the results 
for course grade, the results for continuation revealed an effect of 
timing. There was a positive main effect of UV condition for the 
third essay, Wald = 4.38, B = .25, p = .036, indicating that 


Table 5 


students who wrote a UV essay in the final unit of the course were 
more likely to enroll in the second biology course than students 
who wrote a control essay in the final unit of course. However, this 
effect was qualified by a significant two-way interaction between 
the third essay and prior GPA, Wald = 4.48, B = .23, p = .034, 
suggesting that UV implemented during the final unit of the course 
was most effective for promoting enrollment for high-performing 
students and may have decreased enrollment for students with very 
low (more than 1 standard deviation below the mean) prior GPAs 
(see Figure 2). 

There was also a significant three-way interaction between the 
three UV condition terms, Wald = 16.96, B = .48, p < .001. To 
probe the meaning of the three-way interaction, we conducted post 
hoc comparisons using LSD (see Table 4) and discuss general 
patterns below. There were positive main effects of prior GPA, 
Wald = 14.84, B = .41, p < .001, and initial interest in biology, 
Wald = 27.06, B = .56, p < .001, indicating that students with 
higher prior GPAs and higher interest at the beginning of the 


Effects of Intervention Timing and Dosage on Course Grade, Continuation, and Articulated 


Utility Value 


Continuation to second 


Biology course grade course Articulated utility value 

Measure B (559) Dp B Wald p B (559) Pp 
Essay | (El) OL 30 765 02 03 867 44 17.32. .000 
Essay 2 (E2) .05 1.55 122 —.02 03 .860 4 17.35 .000 
Essay 3 (E3) .03 OT 331 25 4.38 .036 AS 17.57 .000 
El X E2 —.01 .16 877 —.00 00 = .995 05 2.07 = .039 
El X E3 —.04 1.19 233 .04 14.708 —.01 —.22 824 
E2 X E3 .02 48 631 10 69 405 —.01 —50 621 
El X E2 X E3 .09 2.78 006 48 16.96  .000 .00 O1 99 
Prior GPA 61 18.45 .000 Al 14.84 .000 .04 1.53 .126 
El X GPA —.01 .33 742 —.05 22 641 =01 —43 665 
E2 X GPA —.02 50 619 .06 35 555 05 1.84  .066 
E3 X GPA 02 56 573 23 4.48 034 05 1.92 .056 
El X E2 X GPA .04 1.13 258 lil 1.10  .295 O01 35.730 
El X E3 X GPA .02 72 A71 —.02 03 861 .00 16.874 
E2 X E3 X GPA —.06 1.80 072 —.07 38 538 03 1.09 = .275 
El X E2 X E3 X GPA =,02 AT 641 12 1.23 .267 =,01 —.28 — .780 
Gender .10 2.89 .004 —.16 2.06 152 —.03 1.08 ~ .281 
Initial interest in biology 12 3.48 001 56 27.06 .000 02 71 478 


Note. 
GPA = grade-point average. 


Essay 1, Essay 2, and Essay 3 (control = —1, utility value = 1); Gender (female = —1, male = 1). 
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Probability of Continuation as a Function of Essay 3 and Prior GPA 
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Figure 2. Probability of continuing to the second course as a function of 
essay 3 (Control or Utility Value) and prior GPA. 


course were more likely to enroll in the second course. There was 
no effect of gender. 

Post hoc analyses of the three-way interaction on course grades 
and continuation. For performance, a dosage pattern emerged 
among the UV conditions based on post hoc comparisons using 
LSD (see Table 4). In general, both three doses as well as a single 
dose of UV (at any time point) promoted performance, compared 
to the control condition (see Figure 3). Although the condition in 


3.2 


2.8 


2.6 


Biology Course Grade 


2.4 


2.2 


ccc vec cvc cCV we vCv CV VW 


Figure 3. Biology course grade as a function of experimental condition. 
Condition labels reflect the order in which students completed the writing 
assignment (C = Control, V = Utility Value); therefore, consonant- 
vowelconsonant (CVC) refers to the condition in which students wrote a 
control essay in the first unit, a utility value essay in the second unit, and 
a control essay in the third unit. Error bars represent 95% confidence 
intervals. 


which students received the full dosage of UV had the highest 
overall mean, it was not more effective than any of the three 
conditions in which students received a single dose of UV. In other 
words, both single doses of UV and the full dosage of UV were 
more effective for performance than two doses of UV. This pattern 
was not predicted, but may reflect positive effects of novelty with 
a single dose and the predicted positive effect of three doses 
administered throughout the semester. 

The pattern was similar for continuation to the second course. A 
single dose of UV (administered either at the beginning or the end 
of the semester) and three doses of UV promoted continued 
enrollment in the biology sequence (see Figure 4). Indeed, 85.92% 
of students who were assigned one UV essay at the beginning of 
the semester, 84.51% of students who were assigned one UV at the 
end of the semester, and 88.16% of students who were assigned all 
three UV essays continued to the next semester, compared to only 
69.51% in the control condition. Although the condition in which 
students received the full dosage of UV had the highest continu- 
ation rate, it was not more effective at promoting continued en- 
rollment than a single dose of UV administered either at the 
beginning or the end of the semester. For both performance and 
continuation, this pattern suggests that either one novel dose of UV 
or three consistent doses of UV were particularly effective, 
whereas two doses were not. 

Mediation analyses. We used Hayes’ (2013) bootstrapping 
procedure with PROCESS software, which allowed us to test a 
mediation model similar to Figure 1, with biology course grade as 
a mediator of the three-way UV interaction on continuation to the 
second course of the sequence. We hypothesized that the full 
dosage of the UV intervention would increase grades in the course, 
leading students to be more likely to enroll in the second course of 
the sequence. Results based on 5,000 bootstrapped samples indi- 
cate that the indirect effect via biology course grade equaled .06, 
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Figure 4. Percentage of students who continued to the second course of 
the biology sequence as a function of experimental condition. Condition 
labels reflect the order in which students completed the writing assignment 
(C = Control, V = Utility Value); therefore, CVC refers to the condition 
in which students wrote a control essay in the first unit, a utility value essay 
in the second unit, and a control essay in the third unit. Error bars represent 
95% confidence intervals. 
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95% CI [.0164, .1268]. The fact that zero falls outside this interval 
indicates significant partial mediation. 

We also tested whether articulated utility value (measured be- 
fore the end of the course) explained the three-way UV interaction 
on course grades and continuation; however, neither indirect effect 
was significant. The indirect effect via articulated utility value 
equaled = .00, 95% CI [—.0115, .0161] for biology course grade, 
and = —.00, 95% CI [—.0043, .0026] for continuation to the 
second course. 


Supplementary Analyses 


Supplementary analysis of dosage effects on articulated 
value. To examine dosage effects on articulated utility value, it 
is important to consider that some students wrote more UV essays 
than others, given our experimental design. Students who received 
one or two UV assignments also received one or two control 
assignments by design, in which we would expect lower levels of 
articulated value, and it is therefore important to examine the 
average level of utility value articulated in UV essays. We con- 
ducted an internal analysis of the UV essays to examine whether 
the dosage of UV assignments contributed to the quality of the UV 
connections students articulated in their essays. Specifically, we 
averaged the level of articulated utility value in all UV essays 
written by each student, but excluded the control essays. For 
instance, if a student was assigned a UV essay for the first two 
units and a control essay for the third unit, we averaged the level 
of articulated utility value in just the first two essays. We tested 
whether the number of UV assignments students received (1, 2, or 
3) predicted the average level of articulated utility value for the 
UV essays written. These analyses exclude the 82 participants who 
wrote three control essays, and thus no UV essays. We also tested 
the interaction with prior GPA, and included gender (fe- 
male = —1, male = 1) and initial interest in biology as covariates. 

We found that number of UV essay assignments significantly 
predicted the average level of articulated utility value in UV 
essays, (489) = 2.03, B = .09, p = .043 (M, yy = 2.87, SD, yy = 
1.22; Mayv, = 2.99, SDayv, = -95; Mayy, = 3.14, SDayy, = 
.67). In other words, students articulated higher quality UV con- 
nections if they wrote more UV essays, suggesting that practice 
helps students find better UV connections. There was a significant 
effect of gender, (489) = 2.35, B = —.11, p = .019, indicating 
that the average level of articulated utility value in UV essays was 
higher among women than among men. Initial interest in biology 
was also a significant predictor of the average level of articulated 
utility value, 7(489) = 1.99, B = .09, p = .047. Prior GPA and the 
interaction with number of UV essays were not significant. 

To test whether there were “spillover” effects of the UV essay 
assignments, we examined whether the dosage of UV assignments 
predicted the level of articulated utility value in all control essays 
written by each student, excluding all UV essays. For instance, if 
a student was assigned a UV essay for the first two units and a 
control essay for the third unit, the student might spontaneously 
generate UV connections in the control essay. Using the same 
model as above, we tested whether the number of UV assignments 
students received (0, 1, or 2) predicted the average level of artic- 
ulated utility value for the control essays written. These analyses 
exclude the 76 participants who wrote three UV essays, and thus 
no control essays. 


We found that number of UV essay assignments was a marginal 
predictor of the average level of articulated utility value in control 
essays, t(495) = 1.91, B = .08, p = .057 (Mouy = -45, SDouy = 
40; Migy = .50, SDigy = 57; Magy, = 59, SDoyy, = -81). 
Though this effect did not reach statistic significant, it was in the 
hypothesized direction. In other words, students who wrote more 
UV essays articulated somewhat more UV connections in their 
control essays, on average, indicating that writing UV essays may 
increase the likelihood of making UV connections in subsequent 
control essays. Gender, initial interest in biology, Prior GPA and 
the interaction with number of UV essays were not significant. 

Supplementary analysis of timing. To examine the effect of 
timing more directly, we conducted an exploratory analysis in a 
3-cell subset of the experimental design. We identified the three 
UV conditions in which students received a single UV assignment, 
combined with two control assignments (UV first, UV second, or 
UV third). There was not enough variability in STEM major 
persistence within these three cells to test persistence as an out- 
come. Therefore, we examined the effects of timing on course 
grade, continuation to the second course, and articulated utility 
value. We hypothesized that writing a UV essay at the beginning 
of the semester (UV first) would be more effective than writing a 
UV essay at the end of the semester (UV third), especially for 
low-achieving students. To test our hypothesis, we used two 
dummy variables to examine the effects of the three conditions on 
course grade and continuation. The first dummy variable tested our 
hypothesis directly and compared students who wrote a UV essay 
in the first unit, followed by two control essays, versus students 
who wrote a UV essay in the third unit, preceded by two control 
essays (UV first = 1, UV third = 0). The second dummy variable 
compared students who wrote a UV essay in the second unit versus 
students who wrote a UV essay in the third unit (UV second = 1, 
UV third = 0). We interacted the two dummy variables with prior 
GPA and included gender (female = —1, male = 1) and initial 
interest in biology as covariates.” 

There was a significant two-way interaction on course grade 
between the first dummy variable and prior GPA, 7(204) = 2.15, 
8B = —.17, p = .033. This shows that students with low prior GPAs 
earned higher grades in the course when they wrote about UV in 
the first unit (Y = 2.68), compared with writing a UV essay in the 
third unit (Y = 2.49). The opposite was true for students with high 
prior GPAs, in that writing a UV essay in the third unit was more 
beneficial (Y = 3.51) than writing a UV essay in the first unit 
(Y = 3.33). This interaction was in the same direction for contin- 
uation, though the effect did not reach statistical significance, 
Wald = 3.47, B .86, p = .063. There were no significant main 
effects or interactions on articulated utility value. As noted in the 
primary analyses, prior GPA was a significant positive predictor of 
biology course grade, #204) = 8.03, B = .77, p < .001, and 
continuation to the second course in the biology sequence, Wald = 
7.85, B = .98, p = .005, and initial interest in biology was a 
significant positive predictor of continuation, Wald = 4.96, B = 
.50, p = .026. There was no effect of gender on course grade as in 


3 In a separate analysis, we compared the three UV conditions in which 
students wrote two UV assignments (UV first & second, UV first & third, 
UV second & third), however no main effects or interactions with prior 
GPA were significant. 
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the primary analyses, or any other outcome, possibly due to the 
reduced sample. 

Taken together, these analyses suggest that students with a 
history of poor performance may benefit from writing a UV essay 
in the beginning of the semester, whereas high-performing stu- 
dents may benefit from writing a UV essay at the end of the 
semester. This is consistent with the effect of the third essay. We 
tested timing in two different ways, once with the fully crossed 
model and once with a subset of the design that examined timing 
across the three UV conditions in which students received a single 
UV assignment. Both tests of timing reveal that a UV assignment 
at the third time point was especially important for students with 
high prior GPAs. 


Summary of Dosage and Timing Effects 
Across Measures 


In our dosage analysis we found that three doses of the UV 
intervention was most beneficial for making high-quality UV 
connections, yet a single dose was just as effective as three doses 
on performance and continuation. In other words, either 1 novel 
UV assignment or 3 consistent UV assignments implemented 
throughout the semester were effective. In contrast, in our timing 
analysis we found that the results for a single dose are complex, 
because the timing is crucial for students depending on their prior 
performance, and the results differ by outcome measure. One dose 
of UV implemented early increased performance for low perform- 
ers, whereas one dose of UV implemented late increased contin- 
uation for high performers. Across all analyses, we found that two 
doses were not effective for performance or continuation. Consid- 
ered together, we conclude that full dosage (three UV assignments) 
is optimal to obtain the highest level of articulated UV, the 
beneficial early (first UV) effect on performance for low perform- 
ers, and the beneficial late (3rd UV) effect on continuation for high 
performers. 


Discussion 


The results presented here indicate that a UV writing interven- 
tion implemented in an introductory course can impact perfor- 
mance, continued enrollment, and the retention of students in 
STEM majors. To our knowledge, this is the first study to dem- 
onstrate the downstream implications of a UV writing intervention. 
We found that the UV intervention increased performance in the 
course, as well as persistence in the two-semester biology se- 
quence. Connecting course material to personal goals and values 
influenced whether students decided to enroll in the second course 
of the sequence and this process was partially mediated by their 
grade in the course. In other words, earning a good grade in this 
foundational course provided the springboard for enrolling in the 
second course. In addition to increasing grades and subsequent 
enrollment, our results suggest that the UV intervention also in- 
creased STEM major persistence across the semester. Students 
were more likely to report staying in their STEM major at the end 
of the semester in UV conditions, compared to the control group. 
Although these effects were relatively small in magnitude, it is 
worth noting that the positive effects of the UV intervention were 
robust, given that prior performance was controlled in all analyses. 
Moreover, the control condition was designed to provide pedagog- 


ical value (i.e., students were integrating and summarizing course 
material); even a small increase in performance, retention, and 
major persistence relative to this active control is a noteworthy 
improvement and has real-life implications for students’ academic 
and professional careers. 

Expectancy-value theory posits that perceptions of utility value 
in one context can have downstream implications for other con- 
texts or future goals (Eccles & Wigfield, 2002); however, this has 
not been tested experimentally. Our results show that a UV ma- 
nipulation cannot only improve grades in the immediate context, 
but also improve retention outcomes that are related to students’ 
personal goals, such as sticking with their intended STEM major. 
This finding supports the notion that the connection between UV 
and future goals is borne out in college contexts, such that a UV 
intervention can impact important decisions that students make 
about their academic and career paths. 


Implications for Intervention Dosage 


The design of this study and the large sample size allowed us to 
test effects of dosage and timing, to determine how best to imple- 
ment the intervention. We found that assigning a single dose of UV 
in any unit was as effective as three UV essays consistently 
throughout the semester (once for each unit of the course) at 
increasing performance and continuation to the second course 
essay. Although students assigned the maximum dosage had the 
highest overall mean grades and continued enrollment rates, as- 
signing only one UV essay had similar effects. In other words, a 
single UV assignment might be powerful because of its novelty 
and three UV assignments might be powerful because of their 
consistency. 

One hypothesis is that students need practice with the UV 
writing assignment in order for it to be effective, and consistency 
when implementing UV assignments might facilitate this. It could 
be that after the first time students attempt to articulate how the 
course material is relevant or useful to them, they are better able to 
generate personal connections a second or third time. Our analysis 
of articulated utility value was consistent with this possibility. Post 
hoc comparisons and our internal analysis of the UV essays 
revealed that there was an additive effect of the UV writing 
assignments on articulated utility value, such that the more UV 
essays students wrote, the higher the quality of their articulated 
utility-value connections, on average. The first UV writing assign- 
ment may prime students to start thinking about utility value for 
other units in the course and the second and third assignments act 
as intervention boosters. In other words, once students write a UV 
essay, it may be easier to write additional UV essays, and thus they 
can generate higher-quality UV connections in subsequent essays. 

Interestingly, although the UV intervention promoted articulated 
utility value (as intended) it did not mediate course grades or the 
long-term impact of the intervention. It is possible that students 
may think about or engage with the material differently as a result 
of repeatedly writing UV essays, resulting in increased perfor- 
mance in the class, and it may not necessarily be the quality of UV 
connections that drives performance. For instance, the UV assign- 
ments may set into motion other psychological processes and 
behaviors that influenced performance, such as greater engage- 
ment with course materials more broadly, more class participation, 
or feeling a sense of purpose. It will be important in future research 
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to explore the underlying psychological changes and behaviors 
that occur as a result of this intervention that were not assessed in 
the current study. 

An important feature of the UV intervention, as implemented in 
the current study, is that students received feedback from graders. 
Indeed, if UV assignments are to be integrated into course curri- 
cula at scale, it is important to test them in the way that writing 
assignments are typically utilized in science classes, which usually 
involves grading for adherence to instructions and content mastery. 
With such feedback, students can have more than one opportunity 
to revise their essay strategy. For instance, if in the first UV essay, 
a student makes only a superficial connection to the material (e.g., 
“carbohydrates are important because humans need them to sur- 
vive”), rather than a well-developed, personal connection to the 
material (e.g., “As a runner, I can use what I learned about 
carbohydrates in this class to create my diet plan and optimally 
train for a marathon”), the grader can encourage the student to 
generate more specific and personal connections. The benefits of 
this correction process may not occur if the student only completes 
one or two essay assignments and therefore, multiple essay assign- 
ments might be needed to optimize this process. Though the UV 
exercise requires instructors to invest time in grading the essays, 
we believe this cost is justified given that students benefit from the 
intervention not only in terms of increased course grades, but also 
by persisting in the course sequence and STEM fields more 
broadly. It is unknown whether feedback on the assignment is an 
essential part of the intervention or whether simply completing the 
assignment thoughtfully is sufficient for its impact on student 
motivation and performance; therefore, it is possible that the time 
investment could be reduced by limiting instructor feedback. This 
is an important question, given that the cost of grading may be a 
deterrent for some departments to add UV writing to their curri- 
cula, and it should be addressed in future research. 


Implications for Intervention Timing 


In our primary analyses of intervention efficacy, we did not 
replicate previous research that showed that the UV intervention 
was most effective for students with a history of poor performance 
(Harackiewicz et al., 2016; Hulleman et al., 2010). Instead, it was 
not until we conducted a detailed analysis of intervention timing 
that we discovered when interactions with prior performance were 
likely to emerge. We found that students with high prior GPAs 
were more likely to continue to the second course if they wrote a 
UV essay at the end of the semester. Previous tests of the UV 
writing intervention had not found positive effects for high- 
performing students, but this may be due to the fact that no study 
has examined outcomes that extend beyond the course in which the 
intervention was implemented. In this study, we examined contin- 
uation and found a positive effect for high-performing students. 

It is possible that the timing of the third essay coincided with the 
timing of when students were deciding which courses to take the 
following semester. Most students in this course were eligible to 
enroll in courses for the spring semester in the last four weeks of 
the fall semester, and the third writing assignment was imple- 
mented during this period. High-performing students may need 
less support from a UV intervention administered repeatedly 
throughout the semester and may benefit most from a UV writing 
assignment when it is implemented at a critical time in the 


decision-making process. The third UV essay may have given 
these students that extra “nudge” to enroll in the next course. 
Interestingly, the third UV essay may have been detrimental for 
students with very low prior GPAs, in that they were less likely to 
continue to the second course after writing a UV essay in the final 
unit of the course. This is consistent with the notion that enhancing 
value with students who do not feel competent could actually lead 
to decreases in long-term motivation (e.g., Canning & Harackie- 
wicz, 2015; Trautwein et al., 2012). Indeed, expectancy-value 
theory suggests that when students have higher value for an 
academic domain, but low expectations of being able to succeed in 
that domain, the value will only do so much to push them to 
continue— but if they have high value for the domain paired with 
high expectations to succeed in the domain, they will be more 
likely to continue (Eccles & Wigfield, 2002; Nagengast et al., 
2011). Our results support this positive interaction between expec- 
tancies (prior performance) and value (utility value) on continued 
motivation. Students with high expectancies (those with high prior 
performance) benefitted more from an injection of value (the UV 
intervention) at the end of the semester when they were making 
their enrollment decisions. 

In contrast, when we compared early versus late implementation 
of the UV intervention, students with a history of poor perfor- 
mance needed the UV intervention early in the course for it to 
affect their performance in the course. Our internal analysis of 
intervention timing suggests that if only one UV assignment is 
given, students with a history of poor performance benefit more in 
terms of performance when the UV intervention is implemented 
early in the semester, whereas high-performing students benefit 
more when the UV intervention is implemented in the last (vs. 
first) unit of the course. Thus, we tested timing in two different 
ways, once with the fully crossed model and once with a subset of 
the design that examined timing across the three UV conditions in 
which students received a single UV assignment. Both tests of 
timing reveal that a UV assignment at the third time point was 
especially important for students with high prior GPAs, whereas 
the analysis in which we compared only a single dose suggests that 
the first time point is especially important for increasing perfor- 
mance for students with low prior GPAs. 

This is consistent with prior research, in that the UV interven- 
tion was most effective for lower performing students, but only 
when they complete a UV assignment early in the semester (Hul- 
leman et al., 2017). Our results indicated that the timing of the UV 
intervention may be particularly important for students, depending 
on their prior performance, and this has implication for replicating 
past research. Because the UV intervention can facilitate the learn- 
ing process, students with a history of poor performance may 
benefit most in terms of performance from receiving the interven- 
tion early in the semester, to start them off on the right foot. In 
contrast, students with a history of high performance (who are 
already more likely to perform well in the course) may benefit 
most from an intervention at the end of the semester, to give them 
the extra push of motivation to do well in the “home stretch.” 


Recommendations for Practice 


Our results suggest that there is not a one size fits all approach 
to implementing the UV intervention. If educators are concerned 
about students making high-quality UV connections, assigning 
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three doses of UV is optimal. If educators are concerned about 
increasing performance for students struggling in the course, as- 
signing one essay early in the semester may be most effective. If 
educators are concerned about retention, assigning one essay at the 
end of the semester may be most effective, but only for students 
with high prior performance. Naturally, a compromise might be to 
assign two essays, one at the beginning and one at the end; 
however, we found that this condition was not more effective than 
the control on either performance or continuation. Therefore, the 
safest approach in our view is to implement the intervention 
multiple times throughout the semester. In the present study that 
meant implementing the intervention three times, once for each 
unit of the course. Three doses of UV were necessary to obtain the 
highest levels of articulated value, the beneficial early (first UV) 
effect on performance for low performers, and the beneficial late 
(third UV) effect on continuation for high performers. This rec- 
ommendation is consistent with previous research suggesting that 
students may benefit from multiple opportunities to make UV 
connections with the material (Hulleman et al., 2017). 


Limitations and Future Directions 


One limitation of the current study is the lack of diversity in the 
sample. As noted earlier, the sample in this study is entirely White 
and Asian/Asian American. Studies using convenience samples in 
nondiverse contexts can reveal basic processes and intervention 
mechanisms that may apply to all students (Harackiewicz & Prini- 
ski, 2018; Schwartz, Cheng, Salehi, & Wieman, 2016). More 
critically, it is important to test whether the results on retention 
outcomes found here hold for other, more diverse populations. 
Previous research has shown that the UV intervention, at full 
dosage (i.e., three per semester), can help close achievement gaps 
for underrepresented minority and first-generation college students 
(Harackiewicz, Canning, et al., 2016), suggesting that the UV 
intervention might provide a powerful tool for increasing repre- 
sentation of underrepresented ethnic minority and first-generation 
college students in STEM. The results of current study, conducted 
in the same context, are consistent with previous research, in terms 
of course grades, and go beyond previous research by showing that 
these increases in performance can lead to increases in the likeli- 
hood of persistence in STEM. Harackiewicz and colleagues (2016) 
found that the UV intervention effects on course performance were 
strongest for underrepresented minority and first-generation col- 
lege students, so it is possible that these groups would also exhibit 
the strongest effects on persistence. This possibility will need to be 
tested with a large, diverse sample. 

Another limitation of this study is that the timing of the writing 
assignments is confounded with the topic of the unit in which the 
writing assignment was implemented. The UV writing assign- 
ments were administered once for each unit of the course (cellular 
biology, genetics, and evolutionary biology). Therefore, it is un- 
clear whether the effects of timing we found are attributable to the 
unit in which the intervention was implemented or the timing of 
the intervention within the semester. For instance, it could be that 
writing about evolutionary biology in itself was beneficial for 
students with high prior performance, or as we hypothesize, it 
could be that the third writing assignment, regardless of its topic, 
was beneficial for these students because of its timing. To tease 
apart timing and topic, the unit of the course would need to be 


counterbalanced, which is typically not feasible in a large intro- 
ductory course. This is an area for future research. 

In addition, it will be important in future research to explore the 
underlying psychological changes and behaviors that occur as a 
result of this intervention that were not possible to measure in the 
current study. Future studies should include process measures 
directly after each implementation of the utility value essays. It is 
possible that students may experience an increase in perceived 
utility value, interest, or confidence as a result of the intervention. 
To examine the mechanism, process measures need to be collected 
directly after each essay assignment and before course grades are 
determined, which was not feasible in the current study. These 
issues will need to be addressed further in laboratory studies where 
perceptions can be measured closer in time to the intervention. 
Moreover, future researchers should consider other moderating 
variables of the intervention, such as literacy and writing skills and 
the ability to articulate utility value. As implemented in the current 
study, the UV writing assignments require a high level of concep- 
tual work and writing skills, which might prove more challenging 
for students with less academic preparation. It is unknown whether 
the ability to articulate utility value is an integral part of the 
intervention or whether simply completing the assignment 
thoughtfully is sufficient for its impact. This question should be 
addressed in future research. 

Finally, this study only measured students’ intentions to major 
in STEM as a preliminary indicator of persistence, not actual 
degree completion. Our results suggest that the UV intervention 
made it more likely that students would continue majoring in 
STEM fields at the end of a foundational STEM course; however, 
a follow-up study is warranted to determine whether students 
remained in STEM fields 2-3 years later and through graduation. 
By intervening with students early in their STEM training, it may 
be possible to keep them on track for completing a degree in 
STEM. Continuation within a foundational program of introduc- 
tory STEM courses is just one of many steps required to improve 
STEM retention. 


Conclusion 


Retaining students in STEM fields is a national priority. Our 
research suggests that a UV writing intervention can increase 
performance, STEM course-taking, and intentions to major in 
STEM fields. By helping students appreciate the relevance and 
utility value of course material, we create the potential to improve 
STEM retention rates and increase the number of STEM profes- 
sionals. Consistent with a growing body of research (Gasiewski et 
al., 2012; Ost, 2010; Seymour & Hewitt, 1997), we found that a 
positive experience in a single introductory course can have down- 
stream implications for student persistence in STEM majors. 
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